fix: harden complex parameter model loading - #2678

## Summary
Complete the remaining model-persistence hardening with fail-closed Java
deserialization, type-specific policies, bounded metadata and path handling,
session-scoped trust, safe DataType JSON persistence, and explicit trusted
compatibility for legacy native Spark Pipeline artifacts.

## Prompting Intent
Investigate the remaining MSRC recommendation after the earlier public fix,
determine whether unrestricted ComplexParam deserialization was still
reachable, remediate the complete persistence boundary, preserve an explicit
trusted-legacy path where safe migration is impractical, and provide extensive
local regression coverage without exposing private incident details.

## Linked Sources
- Prior public remediation: microsoft#2513
- Review evidence: reviews/serializer-deserialization/task-2513-attempt-1-review-1-gpt-5.6-sol.md

## Rationale
Java deserialization callbacks execute before casts or parameter validation,
and closure-bearing parameters cannot be secured with broad package
allowlists. The implementation therefore defaults to non-executable formats
or narrowly constrained object graphs, fails closed for arbitrary legacy
payloads, and requires an explicit trusted scope for compatibility. Shared
path, graph, stream, and metadata budgets address traversal, aliasing,
compression, and resource-exhaustion risks across nested model graphs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary
Clarify class-policy rejection messaging, remove machine-local paths from the
review evidence, and isolate new UDF test imports so the security patch replays
cleanly onto the spark4.1 compatibility branch.

## Prompting Intent
Resolve every current-head pull-request comment and the release-branch
compatibility failure without weakening the deserialization policy or changing
branch-specific Spark 4.1 UDF behavior.

## Linked Sources
- Pull request: microsoft#2678
- Class-policy feedback: microsoft#2678 (comment)
- Review-artifact feedback: microsoft#2678 (comment)
- Failed compatibility build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=232896880

## Rationale
The class policy supports both exact names and package prefixes, so the error
must describe the combined policy. Repo-relative evidence paths avoid leaking
workstation details. Moving imports into the added tests keeps the functional
master change intact while avoiding overlap with spark4.1's branch-only UDF
helper object; a local three-way replay confirmed the complete patch applies
cleanly to the current spark4.1 tip.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary
Express Spark text framing from its explicit newline bytes and rename two
persistence tests so their names identify the parameter type under test.

## Prompting Intent
Resolve the current-head review comments precisely while preserving Spark
3.5's actual text-output framing semantics and keeping the security boundary
tests understandable across master and spark4.1.

## Linked Sources
- Pull request: microsoft#2678
- Framing review: microsoft#2678 (comment)
- Evaluator test review: microsoft#2678 (comment)
- ParamMap test review: microsoft#2678 (comment)
- Spark 3.5 TextOptions: https://github.com/apache/spark/blob/v3.5.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextOptions.scala

## Rationale
Spark's text writer does not use the platform line separator by default; its
TextOptions contract explicitly uses UTF-8 newline bytes. Computing the byte
length from that literal documents the dependency without introducing
incorrect Windows-specific accounting. Accurate test names improve failure
diagnostics without changing coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary
Preserve the requested JVM component type when generic serializers load
PipelineStage arrays, provide fresh stateful resource filters per stream, and
disambiguate Spark Transformer from Scala reflection for Spark 4.1 builds.

## Prompting Intent
Resolve all suppressed current-head review findings and the spark4.1
compatibility compile failure without weakening the secure Pipeline serializer
or its model-wide resource controls.

## Linked Sources
- Pull request: microsoft#2678
- Suppressed current-head review body: microsoft#2678
- Failed spark4.1 build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=232900732

## Rationale
JVM arrays are reified, so returning `Array[PipelineStage]` through a generic
`Serializer[Array[Estimator[_]]]` cast is not type safe. A small adapter now
validates elements and allocates the exact requested component array. Resource
filters track aggregate array bytes and therefore must never be shared across
streams. Fully qualifying Spark's Transformer avoids a Scala 2.13 reflection
name collision while remaining source-compatible with the master baseline.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary
Use overflow-safe multiplication and addition for declared array byte accounting, rejecting arithmetic overflow before it can weaken the deserialization resource budget. Add regressions for both per-array multiplication and aggregate addition overflow.

## Prompting Intent
Resolve the current-head automated review finding on the model deserialization hardening PR while preserving fail-closed JEP 290 resource enforcement and cross-version compatibility.

## Linked Sources
- Pull request: microsoft#2678
- Review comment: microsoft#2678 (comment)
- Prior remediation: microsoft#2513

## Rationale
Exact arithmetic makes overflow an explicit rejection instead of allowing signed Long wraparound. This keeps the stateful aggregate budget monotonic and avoids saturation logic that could obscure malformed stream accounting.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary
Add a fail-closed JEP 290 adapter that selects the Java 9+ java.io API or the Java 8 sun.misc backport at runtime without linking the published Scala sources to either interface. Keep per-stream class, graph, stream, and aggregate array limits active across supported Java versions.

## Prompting Intent
Resolve the exact-head Azure Publish compilation failure under Temurin 8 without weakening the model deserialization hardening or the overflow fix requested by automated review, while retaining Spark 3.5 and spark4.1 compatibility.

## Linked Sources
- Pull request: microsoft#2678
- Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=232908468&view=logs&jobId=0ccccc7a-9630-5914-467b-15a9c61f0287
- Overflow review comment: microsoft#2678 (comment)
- JEP 290: https://openjdk.org/jeps/290

## Rationale
Isolating runtime API differences behind a dynamic proxy preserves Java 8 source compatibility and per-stream JEP 290 enforcement on both API packages. This is safer than dropping resource filtering or changing the publication toolchain; runtimes without either supported filter API fail before reading an object graph.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary
Scope the ArtifactPathResolver class-resource stream with the repository using helper so assertion or read failures cannot leak file or jar handles.

## Prompting Intent
Resolve the current-head suppressed automated review finding without changing the Java 8 linkage assertion or production behavior, and preserve cross-version test compatibility.

## Linked Sources
- Pull request: microsoft#2678
- Automated review head: microsoft@0036b77

## Rationale
Reusing StreamUtilities.using follows the existing test convention and guarantees closure on both success and failure without duplicating manual try/finally cleanup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary
Centralize trusted handling for test-generated model fixtures, bind Python model readers to the active Spark session, correct mismatched test readers, and route security coverage through executed CI shards. Remove the obsolete review artifact and support Scala 2.13 singleton serialization within the constrained LightGBM policy.

## Prompting Intent
The engineer asked to finish the remaining MSRC deserialization remediation as a well-tested pull request, resolve review and CI failures, and reduce the number of changed files without weakening the security boundary.

## Linked Sources
- MSRC incident and remaining-fix assessment: https://portal.microsofticm.com/imp/v5/incidents/details/31000000568481/msrc
- Original partial remediation: microsoft#2513
- Follow-up pull request: microsoft#2678

## Rationale
Central fixture scoping replaces dozens of suite-specific trust flags while keeping production defaults fail-closed. Session binding is implemented once in JavaMMLReader rather than duplicated across generated wrappers. Native nested stages use the bounded PipelineSerializer, and Scala 2.13 compatibility permits only the exact serialization proxy while the referenced singleton class remains subject to the per-type allowlist.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary`nRestore Spark ML-compatible metadata part naming for bounded ComplexParams artifacts, align the ModelParam code-generation assertion with PipelineSerializer, and scope the SAR lazy-DataFrame round trip to an explicit trusted session load.

## Prompting Intent`nThe engineer asked to finish the MSRC deserialization remediation as a well-tested pull request, resolve all CI failures, and keep the follow-up narrowly focused instead of adding broad or suite-specific unsafe-deserialization exceptions.

## Linked Sources`n- MSRC incident and remaining-fix assessment: https://portal.microsofticm.com/imp/v5/incidents/details/31000000568481/msrc`n- Original partial remediation: https://github.com/microsoft/SynapseML/pull/2513`n- Follow-up pull request: microsoft#2678

## Rationale`nWriting metadata directly as part-00000 preserves the bounded, session-backed serializer while retaining Spark ML and sparklyr on-disk compatibility; changing individual R suites would only hide the production contract break. The SAR test enables trust only around loading its locally created lazy DataFrame parameters and restores the previous session state, so production remains fail-closed. The remaining Scala change updates a stale assertion rather than altering runtime behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary
Route generated R complex-stage loading through the bounded PipelineSerializer and add a same-thread, session-bound trusted artifact scope for language bindings. Generated Python tests use the scope so MLflow's native PipelineModel persistence remains compatible without making configuration-only native loading trusted.

## Prompting Intent
The engineer asked to complete the remaining MSRC insecure-deserialization mitigation, make PR microsoft#2678 merge-ready, reduce unnecessary file churn, run focused tests across supported Spark baselines, and resolve the deterministic Python and R CI failures without weakening the fail-closed production boundary.

## Linked Sources
- IcM incident: https://portal.microsofticm.com/imp/v5/incidents/details/31000000568481/msrc
- Prior partial fix: microsoft#2513
- Current pull request: microsoft#2678

## Rationale
Py4J and sparklyr cannot enter a Scala closure-based trust helper, while MLflow calls native PipelineModel persistence internally. A closeable token preserves the existing Spark-session, gateway-thread, and aggregate metadata-budget boundaries instead of allowing ambient configuration to authorize native Pipeline loading. Centralizing the R loader on PipelineStageWrappable removes duplicate unsafe ml_load paths and avoids adding more compatibility files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary
Route generated R fixture loading for nested pipeline stages through the bounded PipelineSerializer path while retaining each target branch's existing rLoadLine implementation. This removes the Spark 4.1 patch conflicts and drops PipelineStageParam.scala and TransformerParam.scala from the aggregate PR diff.

## Prompting Intent
The engineer asked to make the MSRC deserialization remediation pull request merge-ready, run the relevant tests, resolve CI failures and review feedback, and reduce unnecessary file churn without weakening the fail-closed production boundary.

## Linked Sources
- IcM incident: https://portal.microsofticm.com/imp/v5/incidents/details/31000000568481/msrc
- Original partial remediation: microsoft#2513
- Remediation pull request: microsoft#2678
- Azure validation checks: https://github.com/microsoft/SynapseML/pull/2678/checks

## Rationale
The Spark 4.1 branch intentionally uses a different sparklyr extraction expression, so changing rLoadLine on master created textual port conflicts despite equivalent behavior. Selecting the bounded loader at the sole R fixture-generation call site preserves branch-specific source and public APIs, keeps generated tests on PipelineSerializer with an explicit Spark session, and makes the complete patch apply cleanly to Spark 4.1.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>